Skip to content

feat(ci): coverage regression gate vs merge-base - #1036

Open
AlemTuzlak wants to merge 10 commits into
mainfrom
feat/coverage-reporting
Open

feat(ci): coverage regression gate vs merge-base#1036
AlemTuzlak wants to merge 10 commits into
mainfrom
feat/coverage-reporting

Conversation

@AlemTuzlak

@AlemTuzlak AlemTuzlak commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

CI now fails a PR when coverage in an affected package drops by more than 0.5 percentage points versus that PR's merge-base with main. There is no committed baseline file. The numbers show up on the Coverage job summary (Checks tab), not as a PR comment.

🎯 Changes

Adds a CI-only coverage ratchet:

  1. test:coverage on every public package (Vitest json-summary over src/**).
  2. scripts/coverage-check.mjs compares two runs and fails on a drop.
  3. The Coverage job measures affected packages on the PR and on $NX_BASE, then compares.
  4. .github/workflows/coverage.yml runs test:coverage:all on main to warm the Nx Cloud cache.

The gate is fail-closed:

  • --expect makes collect fail if a named package has no usable summary.
  • A package measured on base and missing on head fails.
  • Empty head fails (docs-only PRs skip the job via if:).
  • Both Nx runs pass --coverage.reporter=json-summary so a merge-base whose script is still vitest run --coverage still writes coverage-summary.json.

Coverage stays out of pnpm test, test:pr, test:ci, and git hooks. If the job says DROP, add tests. There is no number to override.

✅ Checklist

  • I have followed the steps in the Contributing guide.
  • I have tested this code locally with pnpm run test:pr.
  • Docs: I updated docs/ for this change, or this change is not user-facing.
  • Changeset: I added a changeset (pnpm changeset), or this PR does not change a published package.

test:pr was not run in full. pnpm test:maintainer (includes the new comparator tests) and pnpm format passed. This is CI/docs/devDependency-only, so no changeset.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

@vitest/coverage-v8 is already a devDependency. New test:coverage scripts do not change shipped output.

Testing

Commands run

  • pnpm exec vitest run scripts/coverage-check.test.ts (pass)
  • pnpm test:maintainer (pass)
  • pnpm format (pass)
  • pnpm test:pr: skipped (too heavy for this CI-only change)

Manual test

  1. Open the Coverage job on this PR after CI finishes.
  2. Confirm the summary table has real compared rows for packages that already had test:coverage on main (ai, ai-client, ai-mcp), not a full list of new.
  3. Confirm a docs-only path still skips the job when the affected list is empty.

How this PR makes testing easy

scripts/coverage-check.test.ts spawns the CLI against temp dirs and is wired into pnpm test:maintainer / test:pr.

Risk / rollback

If collect is too strict, the Coverage job goes red on missing coverage-summary.json instead of silently passing. Revert the PR to remove the required check. Nx cache for test:coverage is isolated from test:lib.

@AlemTuzlak
AlemTuzlak requested a review from a team as a code owner July 31, 2026 12:46
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Added Vitest coverage commands across packages, a coverage summary comparator, pull request coverage validation against the merge base, a main-branch cache-warming workflow, and updated coverage and lint documentation.

Changes

Coverage regression gate

Layer / File(s) Summary
Package coverage commands
package.json, packages/*/package.json
Added root and package-level coverage scripts. The scripts emit text and JSON summaries and use @vitest/coverage-v8 4.1.10.
Coverage comparison and reporting
scripts/coverage-check.mjs
Added package metric collection, base/head comparison, a 0.5 percentage-point tolerance, formatted output, GitHub step summaries, and regression exit handling.
CI coverage validation and cache warming
.github/workflows/pr.yml, .github/workflows/coverage.yml
Added affected-package coverage checks for pull requests and full coverage runs on main to warm the Nx Cloud cache.
Coverage workflow documentation
CLAUDE.md, CONTRIBUTING.md
Documented CI-only coverage, merge-base comparisons, regression handling, cache warming, limitations, and oxlint commands.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Suggested reviewers: crutchcorn

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the CI coverage regression gate and its comparison against the merge-base.
Description check ✅ Passed The description includes the required sections and clearly documents changes, testing, release impact, and rollback considerations.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/coverage-reporting

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Changeset Version Preview

19 package(s) bumped directly, 32 bumped as dependents.

🟥 Major bumps

Package Version Reason
@tanstack/ai-angular 0.3.1 → 1.0.0 Changeset
@tanstack/ai-durable-stream 0.0.0 → 1.0.0 Changeset
@tanstack/ai-memory 0.0.0 → 1.0.0 Changeset
@tanstack/ai-openai 0.17.1 → 1.0.0 Changeset
@tanstack/ai-openrouter 0.15.10 → 1.0.0 Changeset
@tanstack/ai-persistence 0.0.0 → 1.0.0 Changeset
@tanstack/ai-preact 0.11.1 → 1.0.0 Changeset
@tanstack/ai-react 0.18.1 → 1.0.0 Changeset
@tanstack/ai-sandbox 0.2.4 → 1.0.0 Changeset
@tanstack/ai-solid 0.15.1 → 1.0.0 Changeset
@tanstack/ai-svelte 0.15.1 → 1.0.0 Changeset
@tanstack/ai-vue 0.15.1 → 1.0.0 Changeset
@tanstack/openai-base 0.9.9 → 1.0.0 Changeset
@tanstack/ai-acp 0.2.3 → 1.0.0 Dependent
@tanstack/ai-anthropic 0.16.3 → 1.0.0 Dependent
@tanstack/ai-bedrock 0.1.4 → 1.0.0 Dependent
@tanstack/ai-claude-code 0.2.3 → 1.0.0 Dependent
@tanstack/ai-code-mode 0.3.8 → 1.0.0 Dependent
@tanstack/ai-code-mode-skills 0.3.11 → 1.0.0 Dependent
@tanstack/ai-codex 0.2.3 → 1.0.0 Dependent
@tanstack/ai-elevenlabs 0.2.34 → 1.0.0 Dependent
@tanstack/ai-fal 0.9.12 → 1.0.0 Dependent
@tanstack/ai-gemini 0.20.1 → 1.0.0 Dependent
@tanstack/ai-grok 0.14.9 → 1.0.0 Dependent
@tanstack/ai-grok-build 0.2.3 → 1.0.0 Dependent
@tanstack/ai-groq 0.5.3 → 1.0.0 Dependent
@tanstack/ai-isolate-node 0.1.47 → 1.0.0 Dependent
@tanstack/ai-isolate-quickjs 0.1.47 → 1.0.0 Dependent
@tanstack/ai-mistral 0.2.3 → 1.0.0 Dependent
@tanstack/ai-ollama 0.8.16 → 1.0.0 Dependent
@tanstack/ai-opencode 0.2.3 → 1.0.0 Dependent
@tanstack/ai-react-ui 0.8.15 → 1.0.0 Dependent
@tanstack/ai-sandbox-cloudflare 0.2.4 → 1.0.0 Dependent
@tanstack/ai-sandbox-daytona 0.2.0 → 1.0.0 Dependent
@tanstack/ai-sandbox-docker 0.2.0 → 1.0.0 Dependent
@tanstack/ai-sandbox-local-process 0.2.0 → 1.0.0 Dependent
@tanstack/ai-sandbox-sprites 0.2.1 → 1.0.0 Dependent
@tanstack/ai-sandbox-vercel 0.2.0 → 1.0.0 Dependent
@tanstack/ai-solid-ui 0.7.14 → 1.0.0 Dependent

🟨 Minor bumps

Package Version Reason
@tanstack/ai 0.42.0 → 0.43.0 Changeset
@tanstack/ai-client 0.22.1 → 0.23.0 Changeset
@tanstack/ai-devtools-core 0.4.24 → 0.5.0 Changeset
@tanstack/ai-event-client 0.6.8 → 0.7.0 Changeset
@tanstack/ai-utils 0.3.1 → 0.4.0 Changeset

🟩 Patch bumps

Package Version Reason
@tanstack/ai-mcp 0.2.5 → 0.2.6 Changeset
@tanstack/ai-isolate-cloudflare 0.2.38 → 0.2.39 Dependent
@tanstack/ai-vue-ui 0.2.34 → 0.2.35 Dependent
@tanstack/preact-ai-devtools 0.1.67 → 0.1.68 Dependent
@tanstack/react-ai-devtools 0.2.67 → 0.2.68 Dependent
@tanstack/solid-ai-devtools 0.2.67 → 0.2.68 Dependent
ag-ui 0.0.2 → 0.0.3 Dependent

@nx-cloud

nx-cloud Bot commented Jul 31, 2026

Copy link
Copy Markdown

View your CI Pipeline Execution ↗ for commit 3a6a7f6

Command Status Duration Result
nx run-many --targets=build --exclude=examples/... ✅ Succeeded 1m 54s View ↗

☁️ Nx Cloud last updated this comment at 2026-08-21 10:42:54 UTC

@pkg-pr-new

pkg-pr-new Bot commented Jul 31, 2026

Copy link
Copy Markdown

Open in StackBlitz

@tanstack/ai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai@1036

@tanstack/ai-acp

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-acp@1036

@tanstack/ai-angular

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-angular@1036

@tanstack/ai-anthropic

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-anthropic@1036

@tanstack/ai-bedrock

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-bedrock@1036

@tanstack/ai-byteplus

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-byteplus@1036

@tanstack/ai-claude-code

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-claude-code@1036

@tanstack/ai-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-client@1036

@tanstack/ai-code-mode

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode@1036

@tanstack/ai-code-mode-snippets

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-code-mode-snippets@1036

@tanstack/ai-codex

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-codex@1036

@tanstack/ai-cohere

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-cohere@1036

@tanstack/ai-devtools-core

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-devtools-core@1036

@tanstack/ai-durable-stream

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-durable-stream@1036

@tanstack/ai-elevenlabs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-elevenlabs@1036

@tanstack/ai-event-client

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-event-client@1036

@tanstack/ai-fal

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-fal@1036

@tanstack/ai-gemini

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-gemini@1036

@tanstack/ai-grok

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-grok@1036

@tanstack/ai-grok-build

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-grok-build@1036

@tanstack/ai-groq

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-groq@1036

@tanstack/ai-isolate-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-cloudflare@1036

@tanstack/ai-isolate-daytona

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-daytona@1036

@tanstack/ai-isolate-node

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-node@1036

@tanstack/ai-isolate-quickjs

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-quickjs@1036

@tanstack/ai-isolate-quickjs-bun

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-isolate-quickjs-bun@1036

@tanstack/ai-llmgateway

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-llmgateway@1036

@tanstack/ai-mcp

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-mcp@1036

@tanstack/ai-memory

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-memory@1036

@tanstack/ai-mistral

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-mistral@1036

@tanstack/ai-octane

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-octane@1036

@tanstack/ai-ollama

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-ollama@1036

@tanstack/ai-openai

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openai@1036

@tanstack/ai-opencode

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-opencode@1036

@tanstack/ai-openrouter

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-openrouter@1036

@tanstack/ai-perplexity

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-perplexity@1036

@tanstack/ai-persistence

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-persistence@1036

@tanstack/ai-preact

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-preact@1036

@tanstack/ai-react

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react@1036

@tanstack/ai-react-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-react-ui@1036

@tanstack/ai-sandbox

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox@1036

@tanstack/ai-sandbox-cloudflare

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-cloudflare@1036

@tanstack/ai-sandbox-daytona

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-daytona@1036

@tanstack/ai-sandbox-docker

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-docker@1036

@tanstack/ai-sandbox-local-process

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-local-process@1036

@tanstack/ai-sandbox-sprites

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-sprites@1036

@tanstack/ai-sandbox-vercel

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-sandbox-vercel@1036

@tanstack/ai-solid

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid@1036

@tanstack/ai-solid-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-solid-ui@1036

@tanstack/ai-svelte

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-svelte@1036

@tanstack/ai-utils

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-utils@1036

@tanstack/ai-vercel-gateway

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vercel-gateway@1036

@tanstack/ai-vertex

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vertex@1036

@tanstack/ai-vue

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue@1036

@tanstack/ai-vue-ui

npm i https://pkg.pr.new/TanStack/ai/@tanstack/ai-vue-ui@1036

@tanstack/openai-base

npm i https://pkg.pr.new/TanStack/ai/@tanstack/openai-base@1036

@tanstack/preact-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/preact-ai-devtools@1036

@tanstack/react-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/react-ai-devtools@1036

@tanstack/solid-ai-devtools

npm i https://pkg.pr.new/TanStack/ai/@tanstack/solid-ai-devtools@1036

commit: 3a6a7f6

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@CONTRIBUTING.md`:
- Line 55: Update the lint command in the contributing guide’s command table
from pnpm test:eslint to pnpm test:oxlint, matching the script exposed by the
root package configuration.

In `@packages/ai-acp/package.json`:
- Line 41: Quote the src/** glob in the test:coverage script’s
--coverage.include argument so the shell cannot expand it before Vitest
processes it. Apply this change at packages/ai-acp/package.json:41-41,
packages/ai-angular/package.json:49-49,
packages/ai-anthropic/package.json:56-56, packages/ai-preact/package.json:41-41,
packages/ai-react-ui/package.json:37-37, packages/ai-react/package.json:41-41,
packages/ai-sandbox-cloudflare/package.json:49-49,
packages/ai-sandbox-daytona/package.json:40-40,
packages/ai-sandbox-docker/package.json:40-40,
packages/ai-sandbox-local-process/package.json:39-39, and
packages/ai-sandbox-sprites/package.json:41-41.

In `@packages/ai-codex/package.json`:
- Line 44: Update the test:coverage script in packages/ai-codex/package.json at
lines 44-44, packages/ai-durable-stream/package.json at lines 39-39,
packages/ai-vue/package.json at lines 36-36, packages/ai/package.json at lines
68-69, and packages/openai-base/package.json at lines 39-39 to include
--passWithNoTests in each Vitest coverage command.

In `@packages/ai-mcp/package.json`:
- Around line 59-60: Add the --passWithNoTests flag to each test:coverage script
in packages/ai-mcp/package.json, packages/ai-mistral/package.json,
packages/ai-openai/package.json, packages/ai-opencode/package.json,
packages/ai-openrouter/package.json, and packages/ai-persistence/package.json,
preserving the existing Vitest coverage options.

In `@packages/ai-solid/package.json`:
- Line 68: Align the local vitest declaration with `@vitest/coverage-v8` at exact
version 4.1.10 in packages/ai-solid/package.json:68-68,
packages/ai-bedrock/package.json:49-49,
packages/ai-claude-code/package.json:58-58,
packages/ai-client/package.json:70-70,
packages/ai-code-mode-skills/package.json:87-87,
packages/ai-code-mode/package.json:72-72,
packages/ai-sandbox-vercel/package.json:53-53,
packages/ai-sandbox/package.json:77-77, packages/ai-solid-ui/package.json:73-73,
packages/ai-svelte/package.json:72-72, and packages/ai-utils/package.json:55-55,
ensuring each package keeps both dependencies on the same exact version.

In `@scripts/coverage-check.mjs`:
- Around line 99-102: Update the coverage comparison flow around the !before
branch to fail the command when a measured package lacks a baseline entry, while
preserving success only for packages in the explicit documented exclusion list.
Ensure the failure directs users to run pnpm test:coverage:update to add the
missing baseline entry, and keep the existing additions reporting for permitted
exclusions.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 572be2b3-80e0-4c3b-8c20-febbbdd35918

📥 Commits

Reviewing files that changed from the base of the PR and between 826cfed and 2cef2c1.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (56)
  • .github/workflows/pr.yml
  • CLAUDE.md
  • CONTRIBUTING.md
  • coverage-baseline.json
  • package.json
  • packages/ai-acp/package.json
  • packages/ai-angular/package.json
  • packages/ai-anthropic/package.json
  • packages/ai-bedrock/package.json
  • packages/ai-claude-code/package.json
  • packages/ai-client/package.json
  • packages/ai-code-mode-skills/package.json
  • packages/ai-code-mode/package.json
  • packages/ai-codex/package.json
  • packages/ai-devtools/package.json
  • packages/ai-durable-stream/package.json
  • packages/ai-elevenlabs/package.json
  • packages/ai-event-client/package.json
  • packages/ai-fal/package.json
  • packages/ai-gemini/package.json
  • packages/ai-grok-build/package.json
  • packages/ai-grok/package.json
  • packages/ai-groq/package.json
  • packages/ai-isolate-cloudflare/package.json
  • packages/ai-isolate-node/package.json
  • packages/ai-isolate-quickjs/package.json
  • packages/ai-mcp/package.json
  • packages/ai-memory/package.json
  • packages/ai-mistral/package.json
  • packages/ai-ollama/package.json
  • packages/ai-openai/package.json
  • packages/ai-opencode/package.json
  • packages/ai-openrouter/package.json
  • packages/ai-persistence/package.json
  • packages/ai-preact/package.json
  • packages/ai-react-ui/package.json
  • packages/ai-react/package.json
  • packages/ai-sandbox-cloudflare/package.json
  • packages/ai-sandbox-daytona/package.json
  • packages/ai-sandbox-docker/package.json
  • packages/ai-sandbox-local-process/package.json
  • packages/ai-sandbox-sprites/package.json
  • packages/ai-sandbox-vercel/package.json
  • packages/ai-sandbox/package.json
  • packages/ai-solid-ui/package.json
  • packages/ai-solid/package.json
  • packages/ai-svelte/package.json
  • packages/ai-utils/package.json
  • packages/ai-vue-ui/package.json
  • packages/ai-vue/package.json
  • packages/ai/package.json
  • packages/openai-base/package.json
  • packages/preact-ai-devtools/package.json
  • packages/react-ai-devtools/package.json
  • packages/solid-ai-devtools/package.json
  • scripts/coverage-check.mjs

Comment thread CONTRIBUTING.md Outdated
Comment thread packages/ai-acp/package.json Outdated
Comment thread packages/ai-codex/package.json Outdated
"lint:fix": "oxlint src --type-aware --fix",
"test:build": "publint --strict",
"test:oxlint": "oxlint src --type-aware",
"test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Add --passWithNoTests to test:coverage for consistency with sibling packages. Most packages in this cohort add --passWithNoTests to the test:coverage script (for example packages/ai-devtools/package.json Line 76, packages/ai-elevenlabs/package.json Line 56, packages/ai-vue-ui/package.json Line 38). These five packages omit the flag. Without it, vitest run --coverage exits non-zero when a package has no matching test files, and this can break the CI Coverage job for nx affected runs.

  • packages/ai-codex/package.json#L44-L44: add --passWithNoTests to the test:coverage script.
  • packages/ai-durable-stream/package.json#L39-L39: add --passWithNoTests to the test:coverage script.
  • packages/ai-vue/package.json#L36-L36: add --passWithNoTests to the test:coverage script.
  • packages/ai/package.json#L68-L69: add --passWithNoTests to the test:coverage script.
  • packages/openai-base/package.json#L39-L39: add --passWithNoTests to the test:coverage script.
🔧 Example fix (apply the equivalent to each listed file)
-    "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
+    "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
"test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
📍 Affects 5 files
  • packages/ai-codex/package.json#L44-L44 (this comment)
  • packages/ai-durable-stream/package.json#L39-L39
  • packages/ai-vue/package.json#L36-L36
  • packages/ai/package.json#L68-L69
  • packages/openai-base/package.json#L39-L39
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai-codex/package.json` at line 44, Update the test:coverage script
in packages/ai-codex/package.json at lines 44-44,
packages/ai-durable-stream/package.json at lines 39-39,
packages/ai-vue/package.json at lines 36-36, packages/ai/package.json at lines
68-69, and packages/openai-base/package.json at lines 39-39 to include
--passWithNoTests in each Vitest coverage command.

Comment thread packages/ai-mcp/package.json Outdated
Comment on lines +59 to +60
"test:oxlint": "oxlint src --type-aware",
"test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Inconsistent test:coverage script: missing --passWithNoTests. Several packages copy the test:coverage template without the --passWithNoTests flag that other packages in this PR use (for example packages/ai-isolate-node/package.json, packages/ai-isolate-cloudflare/package.json, packages/ai-isolate-quickjs/package.json, packages/ai-memory/package.json, and packages/ai-ollama/package.json). If any of these packages has no test files, Vitest exits with a failure instead of passing, and the PR Coverage job (pnpm run test:coverage:check) fails for a reason unrelated to actual coverage regressions.

  • packages/ai-mcp/package.json#L59-L60: add --passWithNoTests to the test:coverage script.
  • packages/ai-mistral/package.json#L35-L35: add --passWithNoTests to the test:coverage script.
  • packages/ai-openai/package.json#L47-L47: add --passWithNoTests to the test:coverage script.
  • packages/ai-opencode/package.json#L43-L43: add --passWithNoTests to the test:coverage script.
  • packages/ai-openrouter/package.json#L43-L43: add --passWithNoTests to the test:coverage script.
  • packages/ai-persistence/package.json#L46-L46: add --passWithNoTests to the test:coverage script.
🛠️ Proposed fix (example for packages/ai-mcp/package.json)
-    "test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
+    "test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"test:oxlint": "oxlint src --type-aware",
"test:coverage": "vitest run --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
"test:oxlint": "oxlint src --type-aware",
"test:coverage": "vitest run --passWithNoTests --coverage --coverage.include=src/** --coverage.reporter=text-summary --coverage.reporter=json-summary",
📍 Affects 6 files
  • packages/ai-mcp/package.json#L59-L60 (this comment)
  • packages/ai-mistral/package.json#L35-L35
  • packages/ai-openai/package.json#L47-L47
  • packages/ai-opencode/package.json#L43-L43
  • packages/ai-openrouter/package.json#L43-L43
  • packages/ai-persistence/package.json#L46-L46
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai-mcp/package.json` around lines 59 - 60, Add the --passWithNoTests
flag to each test:coverage script in packages/ai-mcp/package.json,
packages/ai-mistral/package.json, packages/ai-openai/package.json,
packages/ai-opencode/package.json, packages/ai-openrouter/package.json, and
packages/ai-persistence/package.json, preserving the existing Vitest coverage
options.

"@tanstack/ai": "workspace:*",
"@types/node": "^24.10.1",
"@vitest/coverage-v8": "4.0.14",
"@vitest/coverage-v8": "4.1.10",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

node <<'NODE'
import { execFileSync } from 'node:child_process'
import { readFileSync } from 'node:fs'

const files = execFileSync(
  'find',
  ['packages', '-name', 'package.json', '-type', 'f'],
  { encoding: 'utf8' },
).trim().split('\n').filter(Boolean)

let failed = false

for (const file of files) {
  const pkg = JSON.parse(readFileSync(file, 'utf8'))
  const sections = [
    pkg.dependencies,
    pkg.devDependencies,
    pkg.peerDependencies,
  ]

  const coverage = sections.find(
    section => section?.['`@vitest/coverage-v8`'],
  )?.['`@vitest/coverage-v8`']

  const vitest = sections.find(
    section => section?.vitest,
  )?.vitest

  if (coverage === '4.1.10' && vitest !== '4.1.10') {
    console.error(`${file}: coverage=${coverage}, vitest=${vitest}`)
    failed = true
  }
}

process.exit(failed ? 1 : 0)
NODE

Repository: TanStack/ai

Length of output: 3575


🌐 Web query:

unpkg @vitest/coverage-v8@4.1.10 package.json peerDependencies vitest

💡 Result:

In the @vitest/coverage-v8 version 4.1.10 package.json file, the peer dependency configuration is defined as follows: "peerDependencies": { "vitest": "4.1.10" }, "peerDependenciesMeta": { "vitest": { "optional": false } } This indicates that @vitest/coverage-v8@4.1.10 requires vitest version 4.1.10 as a peer dependency [1][2]. Note that while some users have discussed adding coverage providers as optional peer dependencies to the core vitest package to improve compatibility with strict package managers like pnpm, the @vitest/coverage-v8 package itself strictly requires the corresponding version of vitest [3].

Citations:


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "Root package manifests with vitest declarations:"
rg -n --glob 'package.json' '"vitest"\s*:' .

echo
echo "Top-level lockfile summary for vitest and coverage-v8:"
if [ -f pnpm-lock.yaml ]; then
  rg -n -A 4 -B 1 '(@@ )?/(vitest|`@vitest/coverage-v8`)(@[0-9.]+)?:?|  vitest@|  `@vitest/coverage-v8`@' pnpm-lock.yaml | head -200
fi

echo
echo "Package files matching original affected names with vitest and coverage declarations:"
for f in packages/ai-solid/package.json packages/ai-bedrock/package.json packages/ai-claude-code/package.json packages/ai-client/package.json packages/ai-code-mode-skills/package.json packages/ai-code-mode/package.json packages/ai-sandbox-vercel/package.json packages/ai-sandbox/package.json packages/ai-solid-ui/package.json packages/ai-svelte/package.json packages/ai-utils/package.json; do
  if [ -f "$f" ]; then
    echo "--- $f"
    node - <<EOF
const fs = require('fs')
const p = JSON.parse(fs.readFileSync(process.argv[1], 'utf8'))
for (const section of ['dependencies','devDependencies','peerDependencies']) {
  if (p[section]) {
    if (p[section].vitest || p.section?.['`@vitest/coverage-v8`']) {
      console.log(section + ':', {
        vitest: p[section].vitest,
        coverage: p[section]['`@vitest/coverage-v8`']
      })
    }
  }
}
EOF
  fi
done

Repository: TanStack/ai

Length of output: 3955


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

echo "Manifest snippets for original affected packages:"
for f in packages/ai-solid/package.json packages/ai-bedrock/package.json packages/ai-claude-code/package.json packages/ai-client/package.json packages/ai-code-mode-skills/package.json packages/ai-code-mode/package.json packages/ai-sandbox-vercel/package.json packages/ai-sandbox/package.json packages/ai-solid-ui/package.json packages/ai-svelte/package.json packages/ai-utils/package.json; do
  if [ -f "$f" ]; then
    echo "--- $f"
    python3 - <<PY
import json, sys
p = json.load(open(sys.argv[1], 'r'))
for section in ['dependencies','devDependencies','peerDependencies']:
    s = p.get(section)
    if s and ('vitest' in s or '`@vitest/coverage-v8`' in s):
        print(section, {
            'vitest': s.get('vitest'),
            '`@vitest/coverage-v8`': s.get('`@vitest/coverage-v8`')
        })
PY
  else
    echo "MISSING $f"
  fi
done

echo
echo "pnpm coverage-v8 declarations:"
rg -n '`@vitest/coverage-v8`|vitest:' pnpm-lock.yaml | head -120

Repository: TanStack/ai

Length of output: 343


🌐 Web query:

@vitest/coverage-v8@4.1.10 peerDependencies vitest exact 4.1.10 pnpm optional peer dependencies optional false

💡 Result:

In the Vitest ecosystem, @vitest/coverage-v8 is declared as an optional peer dependency of vitest [1][2]. This change was introduced to resolve issues with strict package managers like pnpm, which struggled to resolve the coverage provider dynamically at runtime when it was not explicitly listed in the dependency tree [3][1]. Specifically, Vitest versions including 4.1.10 list @vitest/coverage-v8 within their package.json under peerDependencies, with a corresponding entry in peerDependenciesMeta marking it as optional: true [3][1][2]. Regarding pnpm behavior: 1. When peerDependenciesMeta.optional is set to true, the package manager does not require the dependency to be present for the installation to succeed [4]. 2. By default, pnpm's autoInstallPeers setting (which is true) will attempt to automatically install all peer dependencies, including those marked as optional [5][6]. 3. If you wish to prevent pnpm from automatically installing these optional peers, you can set autoInstallPeers to false in your.npmrc or pnpm-workspace.yaml file [5][6]. 4. Note that there have been reported discrepancies where optional peers are still installed despite autoInstallPeers: false due to specific implementation details in certain pnpm versions, though this is generally considered a configuration or behavioral edge case rather than the standard intent [6][7].

Citations:


🌐 Web query:

vitest coverage provider peer dependencies coverage-v8 compatibility vitest 4.1.10

💡 Result:

In Vitest v4.1.10, the @vitest/coverage-v8 package serves as the primary coverage provider [1][2]. As of recent updates within the v4 release cycle, @vitest/coverage-v8 and @vitest/coverage-istanbul are explicitly configured as optional peer dependencies of vitest [3][4]. This change was implemented to resolve package resolution issues, particularly in environments with strict dependency isolation like pnpm, by allowing the coverage provider to be properly linked while remaining optional [4]. Key compatibility considerations for Vitest v4.1.10 include: 1. Version Synchronicity: While these packages are now optional peer dependencies, it remains critical to keep the version of the coverage provider (@vitest/coverage-v8) in sync with the core vitest package [5]. Historically, strict peer dependency requirements have caused installation failures (e.g., ERESOLVE errors) when versions mismatched [5]. 2. Configuration: The V8 provider is the default in Vitest [1][6]. To use it, ensure the package is installed in your project: npm install -D @vitest/coverage-v8 3. Migration Notes: If you are migrating to v4, note that several legacy coverage options (such as coverage.all) have been removed, and the default behavior for coverage report inclusion has changed to only include files loaded during the test run [7][8]. For consistent behavior, ensure that both vitest and @vitest/coverage-v8 are updated to v4.1.10 [9][10].

Citations:


Align vitest with @vitest/coverage-v8 at the point where they are declared.

@vitest/coverage-v8@4.1.10 has an exact vitest peer of 4.1.10. Keep the matching local vitest declaration on the same exact version so updates do not drift the coverage provider from its supported Vitest core version.

📍 Affects 11 files
  • packages/ai-solid/package.json#L68-L68 (this comment)
  • packages/ai-bedrock/package.json#L49-L49
  • packages/ai-claude-code/package.json#L58-L58
  • packages/ai-client/package.json#L70-L70
  • packages/ai-code-mode-skills/package.json#L87-L87
  • packages/ai-code-mode/package.json#L72-L72
  • packages/ai-sandbox-vercel/package.json#L53-L53
  • packages/ai-sandbox/package.json#L77-L77
  • packages/ai-solid-ui/package.json#L73-L73
  • packages/ai-svelte/package.json#L72-L72
  • packages/ai-utils/package.json#L55-L55
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai-solid/package.json` at line 68, Align the local vitest
declaration with `@vitest/coverage-v8` at exact version 4.1.10 in
packages/ai-solid/package.json:68-68, packages/ai-bedrock/package.json:49-49,
packages/ai-claude-code/package.json:58-58,
packages/ai-client/package.json:70-70,
packages/ai-code-mode-skills/package.json:87-87,
packages/ai-code-mode/package.json:72-72,
packages/ai-sandbox-vercel/package.json:53-53,
packages/ai-sandbox/package.json:77-77, packages/ai-solid-ui/package.json:73-73,
packages/ai-svelte/package.json:72-72, and packages/ai-utils/package.json:55-55,
ensuring each package keeps both dependencies on the same exact version.

Comment on lines +99 to +102
if (!before) {
additions.push(name)
rows.push([name, ...METRICS.map((m) => `${after[m].toFixed(2)}%`), 'new'])
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Fail when a measured package has no baseline entry.

Lines 99-102 report an addition but allow the command to succeed. A new or omitted package can therefore pass CI without coverage regression gating.

Treat additions as a failure unless the package is in an explicit documented exclusion list. Require pnpm test:coverage:update to add the baseline entry.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/coverage-check.mjs` around lines 99 - 102, Update the coverage
comparison flow around the !before branch to fail the command when a measured
package lacks a baseline entry, while preserving success only for packages in
the explicit documented exclusion list. Ensure the failure directs users to run
pnpm test:coverage:update to add the missing baseline entry, and keep the
existing additions reporting for permitted exclusions.

@AlemTuzlak

Copy link
Copy Markdown
Contributor Author

Thanks — went through all six. Two applied in 187f107, three declined on the evidence, one design call.

Applied

  • Quote --coverage.include='src/**' (all 50 scripts). Fair point about shell expansion. I checked the quoted form doesn't regress Windows — pnpm normalises it, and ai/ai-mcp/ai-utils re-measured byte-identically (88.66/95.60/77.33) — so it's safe cross-platform.
  • pnpm test:eslintpnpm test:oxlint. Correct, that script doesn't exist. Also fixed the two other stale ESLint mentions in the same file and added the Coverage gate to the list of what CI runs.

Declined — --passWithNoTests on ai, ai-codex, ai-durable-stream, ai-vue, openai-base, ai-mcp, ai-mistral, ai-openai, ai-opencode, ai-openrouter, ai-persistence

Every one of these has tests, so the flag would be actively harmful:

ai: 71 test files          ai-mcp: 13         ai-openrouter: 9
ai-codex: 5                ai-mistral: 1      ai-persistence: 13
ai-durable-stream: 1       ai-openai: 16      openai-base: 10
ai-vue: 5                  ai-opencode: 5

All eleven reported coverage on the green CI run. --passWithNoTests exists only on the packages that genuinely have no test files; adding it here would convert "this package's tests vanished" from a hard failure into a silent pass, which is precisely the regression the coverage gate is meant to catch.

Declined — pinning vitest to exact 4.1.10 in 11 packages

The premise doesn't hold: 45 of the 50 packages don't declare vitest at all (they use the hoisted root dev dependency). The 5 that do — ai-persistence, ai-sandbox, ai-solid, ai-vue — declare ^4.1.10, which resolves to exactly 4.1.10, matching the provider. There is no skew, and pnpm test:sherif (the repo's workspace-consistency gate) passes clean.

Worth noting the underlying concern was real and is what this PR fixes: the provider was pinned to 4.0.14 against vitest 4.1.10, which made coverage unrunnable (ERR_MODULE_NOT_FOUND). That's now aligned.

Design call — failing when a measured package has no baseline entry

Keeping it non-fatal. Hard-failing creates a chicken-and-egg problem: a new package can't be baselined locally (the numbers are Linux-authoritative — see CONTRIBUTING), so the author would need a CI round-trip just to unblock their own PR. The suggested exclusion list is a second config surface that itself needs maintaining.

It isn't silent as-is: unbaselined packages are listed in the run summary, marked new in the table, and their entries are emitted as a paste-ready JSON block. ai-acp and ai-sandbox went through exactly that flow on this PR and it worked. If a package ever does slip, the cost is one PR of unenforced coverage, not a wrong number.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/pr.yml:
- Around line 59-63: The coverage-check.mjs script is new in this PR and does
not exist in the merge-base, causing the command at line 82 to fail when the
workflow checks out NX_BASE. Copy scripts/coverage-check.mjs to a temporary
location in $RUNNER_TEMP before checking out the base revision, then update both
the base-run coverage-check command (around line 82) and the PR-run
coverage-check command (line 63 in the diff) to use the temporary copy path
instead of the local script file. This preserves the head version of the script
for both the base and PR collection runs.

In `@scripts/coverage-check.mjs`:
- Around line 36-38: Update the coverage comparison flow around the zero-total
summary check to distinguish an unmeasured head package from a genuinely
non-comparable summary. When a package resolved from the base intersection has
no head summary or zero statements, fail the gate with the existing “Not
measured on this PR” behavior, unless the package is explicitly excluded;
preserve successful handling for excluded packages and normal comparisons.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fa15c666-6789-4950-b3ff-b4f1e34a62ef

📥 Commits

Reviewing files that changed from the base of the PR and between 187f107 and 90ebdfd.

📒 Files selected for processing (6)
  • .github/workflows/coverage.yml
  • .github/workflows/pr.yml
  • CLAUDE.md
  • CONTRIBUTING.md
  • package.json
  • scripts/coverage-check.mjs
💤 Files with no reviewable changes (1)
  • package.json
🚧 Files skipped from review as they are similar to previous changes (1)
  • CONTRIBUTING.md

Comment thread .github/workflows/pr.yml Outdated
Comment on lines +36 to +38
// A package with no source loaded reports totals of 0/0 as 100%; that is not
// a number worth comparing.
if (summary.total.statements.total === 0) return undefined

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Fail when a previously measured package has no head summary.

The PR workflow resolves package names from the head, then measures the base intersection. Therefore, a name present only in base means that the requested head measurement produced no comparable summary.

A change that disables collection or reduces statements to zero is reported as “Not measured on this PR” and exits successfully. Treat this state as a gate failure, unless the package is in an explicit exclusion list.

Also applies to: 149-175, 212-216

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/coverage-check.mjs` around lines 36 - 38, Update the coverage
comparison flow around the zero-total summary check to distinguish an unmeasured
head package from a genuinely non-comparable summary. When a package resolved
from the base intersection has no head summary or zero statements, fail the gate
with the existing “Not measured on this PR” behavior, unless the package is
explicitly excluded; preserve successful handling for excluded packages and
normal comparisons.

@github-actions github-actions Bot added waiting-on: maintainer The ball is in the maintainers’ court merge-conflicts Conflicts with the base branch — needs a rebase waiting-on: author Waiting for the author to respond or update and removed waiting-on: maintainer The ball is in the maintainers’ court waiting-on: author Waiting for the author to respond or update merge-conflicts Conflicts with the base branch — needs a rebase labels Aug 13, 2026
@github-actions github-actions Bot added waiting-on: author Waiting for the author to respond or update waiting-on: maintainer The ball is in the maintainers’ court merge-conflicts Conflicts with the base branch — needs a rebase and removed waiting-on: maintainer The ball is in the maintainers’ court waiting-on: author Waiting for the author to respond or update merge-conflicts Conflicts with the base branch — needs a rebase labels Aug 18, 2026
@tombeckenham
tombeckenham force-pushed the feat/coverage-reporting branch 2 times, most recently from 238e8e6 to df32eb1 Compare August 20, 2026 10:55
@github-actions github-actions Bot added merge-conflicts Conflicts with the base branch — needs a rebase and removed merge-conflicts Conflicts with the base branch — needs a rebase labels Aug 20, 2026
@tombeckenham tombeckenham added waiting-on: maintainer The ball is in the maintainers’ court and removed waiting-on: author Waiting for the author to respond or update merge-conflicts Conflicts with the base branch — needs a rebase labels Aug 21, 2026
AlemTuzlak and others added 10 commits August 21, 2026 20:31
Adds a `test:coverage` script to every package and a `Coverage` job to the
PR workflow that fails when coverage drops.

Coverage was previously unrunnable: `@vitest/coverage-v8` was pinned to
4.0.14 in all 50 packages while `vitest` resolved to 4.1.10, so vitest could
not resolve the provider at all (`ERR_MODULE_NOT_FOUND`). The two existing
`test:coverage` scripts (ai, ai-client) could not have worked. Aligning the
provider on 4.1.10 fixes it.

- every package gains `test:coverage`, measured over `src/**` with
  json-summary + text-summary reporters (flags live in the script so no
  per-package vitest config changes were needed)
- `scripts/coverage-check.mjs` diffs the fresh summaries against
  `coverage-baseline.json` and exits non-zero on a >0.5pp drop in
  statements/branches/functions/lines; unmeasured packages are skipped
  rather than counted as 0%
- root gains `test:coverage:all`, `test:coverage:check`, `test:coverage:update`
- `coverage-baseline.json` seeded from a full run of the 48 packages whose
  suites pass on this machine
- CONTRIBUTING.md documents the workflow and the two known gaps
The check previously only printed to the CI log, so the numbers were
invisible unless the job went red. Writing the same per-package table with
deltas to $GITHUB_STEP_SUMMARY puts it on the PR's Checks tab on every run,
pass or fail, with no token and no `pull-requests: write` permission.
The first CI run confirmed the gate works — 47 of 50 packages reproduced the
locally generated numbers to the hundredth — and surfaced the three that
could not be measured correctly on Windows:

- ai-acp and ai-sandbox have tests that fail on Windows for reasons unrelated
  to coverage, so they were missing from the baseline entirely
- ai-sandbox-local-process reads ~20pp higher on Windows because its
  process-spawn code branches on platform, which tripped the gate

All three now carry the numbers from the Linux CI run. CONTRIBUTING.md spells
out that the baseline must come from Linux, since CI is what enforces it.
Coverage was already absent from `test`, `test:pr`, `test:ci` and the git
hooks, but the docs still presented it as part of the local workflow — four
coverage rows in CONTRIBUTING's day-to-day command table, and an instruction
to re-baseline with `pnpm test:coverage:update` on your own machine.

That last one was an active trap: the committed numbers are Linux numbers and
a few packages measure differently per platform, so following the doc on a Mac
or Windows box reintroduces exactly the ai-sandbox-local-process failure the
previous commit fixed.

The check now prints a paste-ready JSON block for every package the baseline
needs to absorb, in both the log and the run summary, so re-baselining is a
copy out of CI rather than a local run. `test:coverage:update` stays as the
maintainer escape hatch but is no longer the documented path.
Review feedback:

- quote `--coverage.include='src/**'` in all 50 test:coverage scripts so a
  POSIX shell can't expand the pattern before vitest parses it. Verified the
  quoted form behaves identically on Windows (pnpm normalises it) and yields
  byte-identical percentages, so it is safe cross-platform.
- CONTRIBUTING referenced `pnpm test:eslint`, which does not exist — the repo
  moved to oxlint. Fixed that and the two other stale ESLint mentions, and
  added the Coverage gate to the list of what CI runs.
…ing it

The committed baseline had the maintenance problems it looked like it would:
it needed manual syncing, it was platform-sensitive (ai-sandbox-local-process
reads ~20pp higher on Windows, which broke the gate on this very PR), and
adding or removing a package meant hand-editing a generated file.

The Coverage job now measures each affected package twice in one job — on the
PR head and on its merge-base with main — and compares those. Nothing to keep
in sync, no platform skew (both sides run on the same runner), and package
add/remove needs no action.

- delete coverage-baseline.json, `test:coverage:check`, `test:coverage:update`
  and the paste-a-JSON-block flow; all of it existed only to tend the file
- coverage-check.mjs gains `--collect <dir>` (snapshot summaries so they
  survive the checkout between runs) and `--base <dir> --head <dir>`
- resolve the package list once, with `--with-target=test:coverage`, so both
  sides measure the same set and a docs-only PR resolves to an empty list
  rather than erroring in run-many; the base side intersects that list with
  the projects that exist there, so PR-added packages report as `new`
- new coverage.yml runs test:coverage:all on pushes to main purely to populate
  the Nx Cloud cache: the base-side task hashes then already exist remotely
  and, because test:coverage declares its coverage dir as an output, restore
  without re-running tests

Verified locally: collect/compare round-trip, a drop failing with only the
out-of-tolerance metrics listed, PR-added packages reporting `new` without
failing, removed packages ignored, missing base dir, empty head, bad args.
The merge-base step ran `node scripts/coverage-check.mjs --collect` after
checking out the base commit, where that file does not exist — this PR is what
adds it:

  Error: Cannot find module '/home/runner/work/ai/ai/scripts/coverage-check.mjs'

Not specific to this PR either: any PR that adds or moves the script would hit
the same thing. Copying it to $RUNNER_TEMP before the checkout fixes it and
also guarantees both runs and the comparison use the same version, which the
previous `git checkout $GITHUB_SHA -- scripts/...` in the compare step only
approximated.

The base coverage run itself was fine, at 3/6 Nx cache hits before coverage.yml
has ever run on main.
The previous run passed for the wrong reason. `coverage/` is gitignored, so
checking out the merge base left this PR's coverage files on disk, and the
base-side `--collect` read them back: 47 packages were compared against
themselves, which is why every delta was exactly 0.00.

Both runs now clear `packages/*/coverage` first. Nx restores the directory for
cached tasks, so genuine cache hits still work.

On this PR the base side will now legitimately collect almost nothing — the
merge base predates the json-summary reporter, so there is no base data to
compare against and packages report as `new`. That resolves itself once this
lands and main has the machinery.
zizmor flagged both `${{ steps.affected.outputs.list }}` uses inside run
blocks as code injection via template expansion (pr.yml:69, pr.yml:85), and
it's right: the list is built from package names in the PR's own files, so on
a fork PR a package named with shell metacharacters would be interpolated
straight into the script. Passing it through env keeps it out of the shell
source entirely.

Confirmed clean by running zizmor v1.28.0 locally over both workflows.
Missing summaries were treated as out of scope, so this PR's Coverage
job compared nothing against main. Collect now requires --expect,
packages that vanish from head fail, empty head fails, and both CI
runs pass json-summary so the merge-base writes coverage-summary.json.
@tombeckenham
tombeckenham force-pushed the feat/coverage-reporting branch from df32eb1 to 3a6a7f6 Compare August 21, 2026 10:39
@tombeckenham tombeckenham changed the title feat(ci): coverage reporting with a baseline regression gate feat(ci): coverage regression gate vs merge-base Aug 21, 2026

@tombeckenham tombeckenham left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

The coverage gate is now fail-closed (--expect, vanished packages, empty head), both CI runs force json-summary, the missing packages have test:coverage, and the comparator has spawn tests wired into test:maintainer. Branch is rebased onto main and mergeable.

Watch the Coverage job: it should show real compared rows for ai / ai-client / ai-mcp, not a full list of new.

@github-actions github-actions Bot added waiting-on: author Waiting for the author to respond or update and removed waiting-on: maintainer The ball is in the maintainers’ court labels Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on: author Waiting for the author to respond or update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants